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ABSTRACT 


In fiscal year 1994, the federal government obligated more than $23.5 billion towards 
information technology (IT) products and services, which represents about five percent of 
its total discretionary spending. The Clinger-Cohen Act of 1996 was enacted by the U.S. 
Congress to help prevent wasteful government spending on IT projects by mandating that 
federal agencies develop a process to manage their IT projects as investments, including 
methods to continuously improve the efficiency and effectiveness of their management 
processes. The goal of this study was to develop a model to assess a government 
agency’s IT management processes, specifically the selection, control, and evaluation of 
IT investments. In order to accomplish this, various General Accounting Office (GAO) 
reports were reviewed to determine the best practices being used to manage IT 
investments. Also, a model was developed that can be applied to an agency that already 
has some IT management processes in place. The major finding was that, while the 
critical processes and attributes identified by GAO are helpful in implementing 
improvements to an agency’s processes, each individual agency needs to carefully assess 
the environment in which it operates and choose the management tools and techniques 
that best fit the agency’s vision and mission and its specific environment. 
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L INTRODUCTION 

A. BACKGROUND 

In fiscal year 1994, the federal government obligated more than $23.5 billion 
towards information technology (IT) products and services. This amounted to about five 
percent of the government’s total discretionary spending for the year [Ref l:p.2]. In 
contrast to some private corporations that achieved significant performance 
improvements and costs savings with investments in IT, government investments have 
regularly cost millions more than expected, taken longer to deliver, and failed to deliver 
significant improvements in cost, speed, or quality of service. 

In an effort to ensure government agencies use their IT dollars wisely. Congress 
enacted substantial legislation in the last decade. The Government Performance and 
Results Act of 1993 (GPRA) and the Information Technology Management Reform Act 
of 1996 (later renamed the Clinger-Cohen Act) both aim to improve the long-term 
planning process as well as the day-to-day operations of executive agencies. 

GPRA was designed to improve federal business management, progr am 
effectiveness and public confidence in government by making government offices 
develop measurable and attainable goals and performance measures that could be used to 
measure their progress. This law mandated the development of strategic plans and annual 
target goals including performance measures that would be used to measure progress 
toward attainment of these goals. The results are to be reported annually to Congress and 
are subject to independent audits. 

The Clinger Cohen Act (CCA) specifically tasks agencies to develop a process to 
improve their planning for IT projects to ensure the federal government gets the best 
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possible return for its investment. The Office of Management and Budgeting (0MB) and 
the Chief Information Officer (CIO) from each executive department have issued 
guidance for this planning and are continually trying to improve their investment model. 
Numerous studies of both government and private sector IT investment planning have 
been conducted by the General Accounting Office (GAO), including studies to identify 
the best practices in IT management processes. As the watchdog of Congress, GAO acts 
on the request of a congressional committee or member. Ref 3 was GAO’s first 
investigation into federal IT investment management and issued as a response to a 
request by then Senators Clinger and Cohen. This report directly led to passage of the 
Information Technology Management Reform Act. 

B. PURPOSE 

The purpose of this thesis is twofold. First, to develop a model to evaluate a 
government agency’s IT management processes with regards to their compliance with the 
provisions of Clinger-Cohen Act and to note differences between the processes in-place 
and those of the best practices identified by GAO. Second, to use the model developed in 
this thesis to evaluate the IT management processes of a specific federal agency to 
determine if they comply with the Clinger-Cohen Act and make recommendations to 
improve their processes where applicable. 

C. ASSUMPTIONS AND LIMITATIONS 

Since the Clinger-Cohen Act was implemented as an extension of the 

Government Performance and Results Act, the model developed in this thesis will 

assume that a strategic plan is fully developed and in place for the organizations in 

question. The absence of such a plan will not prevent the assessment of an agency’s IT 

management processes, but the evaluation of the agency’s current processes will be 
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limited to broad comparisons to the model developed by this research. A strategic plan is 
necessary to focus the management policies and reap the maximum benefits envisioned 
by Clinger-Cohen, but neither the agency’s strategic plan, nor their strategic planning 
process will be evaluated in this research. 

The model developed in this thesis will be broad enough to be used as a basis for 
the evaluation of any government agency’s IT investment processes. 0MB and GAO 
issue broad guidance for government agencies, therefore research into the basis of the 
model will be limited to GAO reports and 0MB guidance. This thesis will use academic 
sources to expand some areas to support the audits by GAO or 0MB directives, but GAO 
guidance will form the foundation of the model. 

D. RESEARCH METHODOLOGY 

1. Literature Review 

The methodology will include a literature review of the requirements of the 
Clinger-Cohen Act, associated legislation, and publicly available reports addressing the 
best practices of public and private sector organizations as noted by GAO. Additionally, 
a review of research journals and similar publications in the areas of IT management and 
strategic management will be done to expand the GAO models. 

2. Data Collection 

In order to complete the evaluation of a specific agency’s IT management, a 
review of their existing processes will be completed by studying relevant regulations and 
documented procedures and interviewing personnel involved in the management, 
selection, control, operation, or maintenance of IT investments. 
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E. BENEFITS OF THIS STUDY 

This thesis will provide government managers and IT professionals with an 
insight into the background of current initiatives in IT management reform and provide a 
model to help assess their agency’s IT management processes with reference to the best 
practices identified by GAO. The review of a specific agency provides an additional 
view of managing IT and an example of how each agency needs to evaluate the 
environment they work in and choose the techniques that best fit that environment. 

Additionally this thesis provides the selected agency with an independent, 
objective assessment of the processes they currently use to manage their IT investments 
and provides recommendations for future improvements. 

F. THESIS ORGANIZATION 

This Chapter provides a background into the legislation changing governmental 
IT management, the purpose of this thesis, the assumptions and limitations of the 
research, the methodology of the research, the organization of the thesis, and the benefits 
of this study. 

Chapter II will address the management of IT projects as investments and outline 
the phases of the investment cycle presented by GOA in Ref [3]. The concept of using 
performance measures as a technique to select, control, and evaluate IT investments will 
also be introduced. 

Chapter III will develop a model to compare an agency’s IT management 
processes by adapting the critical processes identified with a mature, effective, and 
efficient IT management plans with the investment cycle described in Chapter II. 
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Chapter IV will use the model in Chapter III to evaluate the compliance with 
CCA and the effectiveness of IT investment management for the White House Situation 
Support Staff, a small federal organization. 

Chapter V will include conclusions regarding IT investment management 
processes and recommendations for further study, both at WHSSS and in the general area 
of IT investment management. 
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II. MANAGING IT PROJECTS AS INVESTMENTS 

Too often, IT projects are viewed as expenses and no analysis is performed to 

measure the expected benefits for the dollars expended. Additionally, accoimting for an 
IT project’s total costs is made difficult by the practice of using procurement and 
operational funds simultaneously or not fully associating all costs with the project. In 
order to ftxlly disclose the funds spent on IT projects and choose the projects that provide 
the best return in benefits realized for the funds spent, a technique to better manage IT 
projects needs to be developed. 

A. THE IT MANAGEMENT PROCESS 

Since the enactment of CCA in 1996, several studies have been completed to 
determine the most efficient management processes for IT projects as well as attempt to 
develop a model that can be applied uniformly and government-wide for ease of review 
and approval of IT Projects. To date, the focus of executive departments in trying to 
comply with CCA has been in the acquisition of large (Department wide) IT projects and 
in trying to establish a Departmental model for each smaller agency to apply to its own 
individual projects. There has been little progress in addressing the questions facing the 
smaller agencies. Specifically, how to choose the investment with the best return and 
how to manage the required processes with a limited number of personnel? 

Another difficult hurdle is transforming the agency’s current IT management 
technique for different projects in various stages of development into the investment 
cycle framework developed by GAO in Ref [3]. After enactment of CCA, GAO 
developed the investment cycle as a model to assist federal agencies in star ting to design 
IT investment management processes. GAO has addressed the transformation problem 

with the Information Technology Investment Management (ITIM) framework in Ref [1]. 
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The ITIM framework is modeled after the Software Engineering Institute’s (SEI) 
framework for assessing an organization’s software management ability [Ref l:p.l39]. 
The ITIM framework uses “levels of maturity” to identify management needs. GAO 
[Ref. 1] not only defines the maturity levels of the framework and their critical aspects, 
but also describes how an agency best transitions from one maturity level to the next. 
This report is useful for understanding an agency’s IT management process (moving 
between levels of maturity) and will be addressed in Chapter III after the groundwork for 
an IT management process is discussed. 

This Chapter will present the common factors of GAO reports [Ref 1 and 2] to 
describe a management process for the control of IT investments. This should be 
applicable to all IT investments regardless of agency size or complexity of the IT project. 
These factors include: the organizational attributes critical to successful IT investments, 
the phases of the IT investment cycle, and IT performance measures. 

B. ATTRIBUTES CRITICAL TO SUCESSFUL IT INVESTMENTS 

While each phase of the management process contains unique requirements to 
ensure the success of the IT investment, an organization must master three distinct 
attnbutes for successful implementation. GAO [Ref. 3] identifies these attributes as: 
senior management involvement, mission focus of projects, and the portfolio approach to 
IT investments. 

1. Senior Management Involvement 

Managers with the authority to purchase, modify, and cancel projects must be 
fully integrated into the IT investment process. Usually these are senior managers. A 
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communication process, both routine meetings for planning and status reporting, and a 
separate technique for reporting problems, must be formally implemented. 

The planning for new investments, resource allocation for the overall IT portfolio, 
and the validation of costs and benefits are made in the budget formulation and 
submission part of the year. The communication process must provide managers, who 
have the authority to modify or cancel IT projects, with the knowledge of which IT 
projects are having problems that are not being corrected at lower levels and are in 
trouble. 

Appropriate review processes, documentation requirements, selection criteria, and 
dollar thresholds to determine the level of management required for decision-making 
must be developed for each type of investment and stage of development. This ensures 
each investment gets the scrutiny warranted by the amount of investment dollars 
involved. The size of the agency and therefore the depth of management layers will 
determine the complexity of implementing senior management involvement. 

2. Mission Focus of Project 

The strategic plan consists of the organization’s mission, goals, and objectives. 
Performance measures must be linked to strategic planning as required by CCA and 
GPRA. This requires developing long-term strategic goals, setting annual organizational 
performance targets in support of those objectives, and annually evaluating performance 
against those targets. 

In order for executive departmental goals to be met, each smaller agency within 
an executive department must ensure that their objectives support their parent agency’s 

goals. This results in the department’s goals being addressed in increasing detail as the 
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process moves down through the organizational structure. Likewise, an IT office’s goals 
need to be expressed in terms describing how they will specifically support their agency’s 
objectives instead of broad terms such as “provide cutting edge IT products.” Without 
sufficiently specific translation of goals, people will not focus on their organization’s true 
mission. 

IT projects must now be selected to help the agency meet their objectives. If an 
IT investment cannot be proven to measurably improve agency mission performance, that 
investment should not be made. Mission benefit, not cost or schedule constraints, must 
be the main measure of success of an IT investment. This point must be stressed in the 
selection of IT investments and will help determine if any corrective actions are needed 
while the IT investment is being developed and deployed. 

The development of the strategic plan and the performance targets is instrumental 
in the implementation of IT investment management since the investments will be chosen 
to help the agency achieve its objectives and mission. The performance measures that IT 
investments are evaluated and ranked by must be focused on the performance measures 
used by the agency’s senior managers to measure progress towards these targets. This is 
also related to the involvement of senior management, as the success of any strategic plan 
is linked to both the involvement and commitment of the agency leadership. 

3. The Portfolio Approach to IT Investments 

Most orgamzations have numerous IT projects in different stages of maturity, 
ranging fi-om new projects to those in the final stages of completion. In order to properly 
control all IT investments, the agency must define a portfolio consisting of all 
investments in every stage of development and of every type. The stages of development 
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include concept exploration through operational use, and must not be confused with the 
phases of the IT investment cycle (which are explained in the next section of this 
chapter), although they are related. Investments that are purely research and development 
should be included in the agency’s R&D portfolio, rather than in its IT investment 
portfolio. Also, different types of IT investments should be identified such as mission 
critical, cross-functional, or administrative. “Mission critical” IT investments help the 
agency accomplish an agency mission (as identified in the agency’s strategic plan). 
“Cross-functional” IT investments address one or more missions and/or one or more 
administrative functions, while “administrative” IT investments address only agency 
administration. 

By building an all-inclusive portfolio of investments, resources can be allocated to 
investments after they are ranked based on their costs and benefits. 

C PHASES OF THE IT INVESTMENT CYCLE 

1. Introduction 

The intrinsic part of the IT management process is the description of the phases of 

the investment cycle as reconunended by GAO [Ref l:p. 3]. Figure 1 displays the three 

phases of the cycle: Selection, Control, and Evaluate. The phases are connected by both 

process and information flow. Data collected from investments in the Control or 

Evaluate phases provides feedback, which can improve decisions made in future 

Selection phase meetings. Likewise, decisions made in the Selection phase to change the 

agency’s portfolio will affect what options are available in the Control phase. While a 

single IT investment will proceed through the cycle in order from Selection to Control to 

Evaluate, the agency’s IT portfolio most likely contains investments in each phase. Thus, 

management is constantly documenting data collected and decisions made in all phases 
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throughout the business cycle. Each of the three phases will be discussed in the 
following sections. 



Figure 1. Phases of the IT Investment Cycle After Ref [1] 


2. Phase One: Selection 

The purpose of the Selection phase is to build the agency’s IT portfolio. This is 
accomplished through three processes: Screening, Ranking, and Allocation. The 
Selection phase, unlike the Control and Evaluate phases, is not ongoing, but normally 
occurs at a distinct point in the agency’s operating cycle. The Selection phase may occur 
annually, quarterly, or at any time during the year. Government agencies may find it 
most useful to conduct the selection phase annually in conjunction with their bu d get 
submission process. 
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fl. The Screening Process 

The GAO describes the screening process as the comparison of all 
investments (potential as well as current investments) against a imiform set of established 
screening criteria and thresholds in order to determine which projects meet the minimum 
requirements for continuing review and at what organizational level the investments must 
be reviewed. The policies of the agency will delineate the thresholds and the 
organizational level of review required for different sized projects. Examples of 
thresholds that would have to be met include a minimum return on investment (ROI), or a 
maximum cost amount based on a percentage of the normal annual IT budget. 

b. The Ranking Process 

The ranking process will assign priorities to the IT investment projects 
that survive the screening process using both measures from the screening process and 
any additional measures needed to breakout each potential investment’s rank or prioritize 
the projects from highest to lowest. In order to accomplish the r anking , several measures 
are necessary. At a minimum, measures of cost vs. benefit (i.e. ROI), risk to the agency, 
and mission accomplishment (i.e. strategic, tactical, or administrative) must be used. The 
measures can then be weighted based on the agency’s abilities and mission. For instance, 
projects addressing a mission that is currently not being met must rank above projects 
aimed at improving other missions even when the ROI is lower or higher risks may be 
involved. Comparisons of the weighted scores for each investment can then be used to 
rank the investments. 

c. The Allocation Process 

The last process in the Selection phase is to allocate funds for the next 

business cycle to investments based on the priorities established in the ranking process. 
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Possible outcomes in the allocation are continuing the funding level of an investment, 
increasing the funding, lowering the funding level, cancellation of an existing investment, 
or deferment of a potential investment due to lack of resources. This final process is 
necessary to allocate the limited funds to the projects that provide the most benefit to the 
agency in the form of mission accomplishment, ROI, and at manageable risk levels. If 
the IT funds were unlimited, all projects could be funded making the ranking and 
allocating processes unnecessary. 


This whole set of processes in the Selection phase is summarized by the 
GAO as follows: 

The Selection phase helps ensure that the organization (1) selects those 
IT projects that will best support mission needs and (2) identifies and 
analyzes a project's risks and proposed benefits before a significant 
amount of project funds are spent. A critical aspect of this phase is 
management understanding and participation and decision-making that is 
driven by accurate, up-to-date data and an emphasis on using IT to 
enhance mission performance. [Ref 2:p.7] 

3. Phase Two: Control 

The purpose of the Control phase is to track the investment thr ou gh its 
development and deployment to ensure the appropriate level of performance is being 
gained from each investment. This helps prevent the loss of resources on investments 
that are not helping the agency improve its strategic mission accomplishment. 


Once investments are selected and funded, they are continuously and consistently 
managed and controlled. The progress of the investment is measured against projected or 
expected cost, schedule, and mission benefits. Performance reviews of investments are 
conducted at key points in the investment’s development. The periodicity of these 
reviews will be set at the beginning of the IT project’s development based on its cost. 
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complexity, or risk and may be altered based on performance during development. High 
cost, complexity, or risk projects require more frequent control. The periodicity of these 
reviews can also be changed as necessary depending on the size, risk, and performance of 
the project during its development and deployment. 

Performance measures used to track the investment will have different thresholds. 
Each investment will have a predicted value (i.e. the “good” or “green” value) of the 
final project for each performance measure. Investments in development or deployment 
with all performance measures in the “green” are progressing as expected. Project 
managers will set warning thresholds (i.e. the “warning” or “yellow” value) at which the 
project manager decides on the corrective action that is required and documents his or her 
decision and any action taken. When the danger threshold (i.e. the “danger” or “red” 
value) is reached, management involvement is increased by trouble reports and the senior 
managers will have to decide whether or not the project needs further corrective action, 
modification, or cancellation based on investment performance and response to corrective 
action taken at lower levels. 

The setting of each of these thresholds is very critical to the management process. 
McGee and Prusak, members of the Ernst & Young consulting firm, stress that 
“orgamzations must treat metrics as attention-focusing devices instead of scorecards” 
[Ref. 4:p.l92]. This must mean that the “yellow” values show situations that require 
attention, and administrators should not be too quick to terminate or even modify 
projects. Likewise, project managers need to imderstand that, even though the 
investment survived the selection process, a point may come where the returns delivered 
by an investment when weighed against the resources allocated to that project may 
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warrant modifications or even termination. This emphasizes the importance of 
documenting corrective actions taken or the basis for not taking action. 

4. Phase Three: Evaluate 

The purpose of the Evaluate phase is to examine the lessons learned from each 
specific project and use these to adjust the processes in the Selection and Control phases 
to continually improve the agency’s management of its IT portfolio. 

After a project has reached its final endpoint (usually full implementation or 
cancellation), a final review is completed. GAO, in its “Guide for Evaluating Agencies’ 
IT Investment Decision-making”, describes three major areas that the final review must 
examine: effect on customers, impact on agency mission, and technical capability. 

The effect on customers” must be examined to determine if the recipients of the 
IT services or users of the IT technology realized the desired benefits. In addition to the 
plaimed benefits, the review must determine the impact of any unplanned aspects of the 
project. This includes negative aspects of the project such as added training or 
inefficiency due to increased project complexity as well as any unplanned benefits that 
may have been realized. This will usually require interviewing or surveying the project’s 
users and maintainers. 

The impact on agency mission” must likewise be examined for both unplanned 
effects on mission as well as whether or not the intended affect on agency mission was 
realized. 

The “technical capability” of the project can be assessed by the trends displayed 
by the history of performance measures during the life of the project (maintained during 

the Control phase) compared with the desired goals expressed in the Selection phase. 
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Other aspects of the technical capability of the project include employee satisfaction with 
the system, the extent to which the system was utilized by employees and customers 
alike, and the expertise displayed by the technicians responsible for the project. 

The most difficult aspect of the final review is to find benefits and/or 
shortcomings that were not necessarily measured by the performance measures tracked in 
the Control phase. Since the goal of the Evaluate phase is to improve the processes in the 
Selection and Control phases, the agency must strive to find which positive or negative 
attributes were realized without being explicitly monitored. Agency managers can then 
modify the Selection and Control of investments to minimize the negative attributes and 
capitalize on the positive. 

Last, the final review procedures must be formalized in agency policy. A 
consistent, thorough procedure must be utilized for all projects. The agency must also be 
able to identify shortcomings in its evaluations and improve on these procedures should 
they find better techniques or documentation to do so. The policy must formalize the 
procedure to be used so that it can be clearly communicated and consistently applied. 
However, policy must be flexible enough to allow continuous improvements to the 
Evaluate phase by allowing application of lessons learned from each experience. 

D. IT PERFORMANCE MEASUREMENTS 

Once the agency understands the IT investment cycle and the basic processes 
required to manage the cycle, a method must be developed to measure the qualities 
(positive and negative) of IT investments. These measurements are used to rank 
investments against each other in the Selection phase and are used to track an 
investment’s progress in the Control phase. A wise selection of “performance 
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measurements” to manage IT investments is crucial to the success of an agency’s IT 
management processes. 

1. Introduction 

As discussed in the Selection and Control phases, consistent and reliable metrics 
must be used when comparing investments or tracking their performance through 
development. There are countless performance measures available for the agency to use, 
but they must be chosen to address three features sought by GAO auditors: repeatability, 
efficiency, and completeness. As seen in Figure 2 this adds a measure of breadth to the 
Select, Control, and Evaluate phases. 

They examine each depth of each phase for the critical features of its performance 
measures, namely repeatability, efficiency, and completeness. A process is “repeatable” 
when the documentation that describes the data required and technique used to make 
decisions allows different personnel to come to the same decision when the process is 
accomplished in another investment cycle. “Efficiency” in a process is achieved when all 
participants in the process fully understand what is expected of the process and both 
wasted time and resources are eliminated from the process. “Completeness” is a direct 
result of the process’ documentation conveying the required inputs and outputs of the 
process such that no data or analyses are missed. 
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Figure 2. Three Dimensions of the IT Investment Evaluation Approach From Ref. [2] 

In addition, as shown in Figure 2, GAO expands the IT management process by 
adding a third dimension consisting of: process, data, and decisions. 

The critical aspect of choosing the appropriate performance measures is 
connecting the investment to the agency mission it supports, and determining what output 
will be measured to track how well the desired outcome is being achieved. The 
importance of identifying the desired outcomes and relating them to measurable outputs 
cannot be stressed enough. The best laid-out strategy will never be realized if the desired 
outcomes are not tied to appropriate outputs that can be measured. These measurements 
are the indicators used to provide feedback for corrective actions in the system to 
continually strive towards the desired outcomes. Chapter 3 will discuss the use of the 
“balanced scorecard” to tie these missions, projects, outputs, and outcomes together. 


19 



The remainder of this chapter will discuss two broad areas that must be measured 
to track IT project performance and finally how these measurements can be compared 
and used to rank projects. GSA’s “Performance Based Management: Eight Steps to 
Develop and Use Information Technology Performance Measures Effectively” discusses 
the measurements of effectiveness, efficiency, and the use of information economics to 
apply these measurements to a scorecard. 

2. Effectiveness 

Effectiveness is the measure of how well an objective is being met. The agency 
must carefully choose the outputs measured and avoid choosing inputs. An example of 
this mistake is an office that tracks the number of requests filled. The number of requests 
filled is tied directly to the number submitted, which is an input into the system not an 
output. A better choice in this case would be the percentage of reports processed within 
two working days. The percentage removes the input (although it is still related to the 
number of inputs) and instead emphasizes the attainment of a business goal, that all 
reports be processed within two working days. 

The issue of effectiveness must be addressed in order to enhance the completeness 
of the agency’s performance measurements. This relates back to the point that no project, 
no matter how well it performs with respect to schedule and cost, should be advanced if it 
does not help the agency achieve its missions. 

3, Efficiency 

Efficiency is the measure of how well a project is providing an output comparing 
benefits to the cost of its operation. The most recognized measurement in this area is 
return on investment (ROI). ROI is the benefits to the agency (measured in dollars) 
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provided by the investment divided by cost of acquiring and operating an investment. 
This calculation is difficult with IT projects because there is usually a myriad of 
intangible benefits that IT investments both deliver and promise to deliver. IT 
corporations have attempted to improve the ROI of IT investments by introducing the 
concept of “enhanced ROI.” The enhanced ROI is no different from the conventional 
ROI calculation except the intangible benefits are given a dollar value. 

The most difficult aspect of the efficiency measurement is the inclusion of 
intangibles. This has to be approached from two perspectives. First, the intangibles must 
be addressed. This may mean starting from a baseline, such as using the IT system 
already in place. Then the intangibles can be measured as incremental changes fi-om the 
baseline. Second, the measurement of intangibles needs to be formalized such that the 
process is consistently applied across all investments in the portfolio, not just projects 
competing for the same mission, since all investments “compete” against each other 
when they are ranked in the Selection phase. 

4. Information Economics 

In order to formalize the measurement of intangibles when dete rmining the value 
of IT investments, the General Services Administration recommends the use of score 
factors, as developed by Parker and Benson [Ref 5]. These factors help to improve the 
ROI calculations by providing factors to assess the value and risks of IT projects. These 
factors can then be used to build a scorecard for each IT investment that will be used to 
rank all IT investments in the portfolio. Agencies can use all or some of the factors, or 
develop their own. 

The Information Economics Scorecard factors [Ref 5:p.53-55] are: 
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Enhanced Return on Investment (ROI)— Assesses the cost-benefit 
analysis plus the benefits created by the IT investment on other parts of 
the organization. Parker and Benson provide techniques for assessing costs 
and benefits of the impact of an IT investment on other departments of the 
organization. They also describe techniques for quantifying the benefits 
associated with increasing the value of a function. For example, electronic 
form processing provides a data entry clerk with the capability to process 
claims, a higher value function. 


Strategic Match (SM) — Assesses the degree to which the proposed 
project corresponds to established agency strategic goals. This factor 
emphasizes the close relationship between IT planning and corporate 
planning and measures the degree to which a potential project contributes 
to the strategy. Projects that are an integral and essential part of the 
corporate strategy receive a higher score than those that are not. Strategic 
Match assesses the extent to which an IT investment enables movement 
towards long-term direction. 


Competitive Advantage (CA) — Assesses the degree to which projects 
create new business opportunities, facilitate business transformation (e.g., 
inter-orgamzation collaboration through electronic commerce), increase 
the agency’s competitiveness or improve the agency’s reputation or image. 
Competitive Advantage requires placing a value on a project’s 
contribution toward achieving one or more of these objectives. 

Management Information (MI) — Assesses a project’s contribution to 
nianagement’s need for information about core activities that involve the 
direct realization of the mission, versus support activities. Measuring a 
project’s conribution to the core activities of the business implies that the 
agency has identified its critical success factors. This measurement is 
obviously subjective because improved management information is 
intangible, but the benefit measurement can be improved if the agency 
first defines those core activities critical to its success, and then selects a 
general strategy to address these issues. 

Le^slatiye Implementation (LI) — Assesses the degree to which the 
project implements legislation. Executive Orders and regulatory 
requirements. For example. Federal law requires INS to process 
passengers arriving at airports from international flights within 45 
minutes. A project receives a high score if it directly implements 
legislation; a moderate score if it indirectly implements legislation; and no 
score if the project does neither. 


Organizational Risk (OR) — Assesses the degree to which an 
information systems project depends on new or untested corporate skill, 
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management capabilities and experience. Although a project may look 
attractive on other dimensions and the technical skills may be available, 
unacceptable risks can exist if other required skills are missing. This does 
not include the technical organization, which will be measured on another 
dimension. Organizational risk also focuses on the extent to which the 
organization is capable of carrying out the changes required by the project, 
that is, the user and business requirements. For example, a high score 
reflects that the business domain organization has no plan for 
implementing the proposed system; management is uncertain about 
responsibility; and processes and procedures have not been documented. 

Strategic IS Architecture (SA) —^Assesses the degree to which the 
proposed project fits into the overall information systems direction and 
conforms to open-system standards. It assumes the existence of a long¬ 
term information systems plan — an architecture or blueprint that provides 
the top-down structure into which future data and systems must fit. 

Definitional Uncertainty (DU) — A negatively weighted factor that 
assesses the degree of specificity of the user’s objectives as communicated 
to the information systems project personnel. Large and complex projects 
that entail extensive software development or require many years to 
deliver have higher risks compared to those projects segmented into 
modules with near-term objectives. 

Technical Uncertainty (TU) — Assesses a project’s dependence on new 
or untried technologies. It may involve one or a combination of several 
new technical skill sets, hardware or software tools. The introduction of an 
untried technology makes a project inherently risky. 

IS Infrastructure Risk (IR) —^Assesses the degree to which the entire IS 
organization is both required to support the project, and prepared to do so. 

It assesses the environment, involving such factors as data administration, 
communications and distributed systems. A project that requires the 
support of many functional areas is inherently more complex and difficult 
to supervise; success may depend on factors outside the direct control of 
the project manager 

Once a numerical score is generated for each category based on a scale defined by 
the agency’s policy, they can be used to build a scorecard by summing the common 
factors together in common groups (financial, benefits, and risk). Scores for financial 
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return, expected benefits, and risk can then be compared for each IT investment to assist 
the ranking in the Selection process. 

An integrated product team (IPT) formed by the Chief Information Officer of the 
Navy developed a simpler information economics scorecard addressing only financial 
costs and gains to calculate a ROI, risks involved, and mission accomplishment [Ref. 
6:ppJ-A-6 to J-A-17]. Whatever scorecard approach is used, the complexity of the 
analyses used needs to fit the size and complexity of both the agency and the investments 
undertaken. 

Finally, the measurements made by information economics scorecards must be 
continually examined in all three phases (Selection, Control, Evaluate) to determine if 
they are useful, and if so, how they can be improved. Since the scorecard is made to 
assist in the ranking of projects in the Selection phase and tracking of investment 
progress in the Control phase, the measurements made by information economics must be 
continually evaluated to ensure they are generating the scores necessary to help the 
agency make sound decisions. 
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III. IMPLEMENTING INFORMATION TECHNOLOGY 
INVESTMENT MANAGEMENT IN SMALL AGENCIES 


A. THE ITIM MATURITY STAGES 
1. Overview 

The previous chapter outlined GAO’s three-phased framework of Select, Control, 
and Evaluate in the IT investment cycle. In “Information Technology Management: A 
Framework for Assessing and Improving Process Maturity,” GAO expanded the three- 
phase framework for IT project management previously promulgated by themselves and 
0MB by adapting the idea of “maturity stages”[Ref 1]. GAO decided on a five-stage 
maturity-based framework based on general industry acceptance of maturity-based 
frameworks (such as the Software Engineering Institute’s copyrighted Capability 
Maturity Model) and GAO’s working experience in applying maturity-based methods in 
IT audits and evaluations [Ref. I:p.l39]. 

Figure 3 illustrates the five stages of increasing maturity along with the “critical 
processes” that are used to characterize each of the stages. The left side of the figure 
shows the wide range of investment management from a narrow project-oriented 
approach to those that encompass enterprise-wide strategic concerns. 

The framework in Figure 3 can be used to identify a specific agency’s IT 
management’s level of ability in one of the five distinct maturity stages. The framework 
is cumulative so that an agency evaluated to operate at a specific maturity stage is 
assumed to be managing the critical processes at this stage as well as the prior stages 
outlined in Figure 3. 
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As noted in Figure 3, agencies at Stage One have little awareness of how their IT 
investments are performing or how they relate to the agency’s mission. The few 
management processes that are in place are generally project oriented and outcomes are 
widely variable. Hence this stage can be described as “undisciplined”. 
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Figure 3. The Five Stages of Maturity Within ITIM From Ref [1] 

At Stage Two agencies begin to implement investment selection and control 
techniques such as business needs identification for IT projects, project oversight, asset 
tracking, and the development of the IT oversight board. These are further enhanced in 
Stage Three by focusing on a comprehensive investment portfolio approach and 
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incorporating portfolio-based benefit and risk criteria linked to near-term business needs 
of the agency. 

Stage Four improves the investment process by instituting post-implementation 
reviews and feedback from these reviews to improve the Selection and Control phases of 
the portfolio. Plus, this stage introduces concern for succeeding information systems and 
technologies that will replace high cost or low value added current investments. 

At the final stage, Stage Five, an agency benchmarks its IT investment processes 
against best practices of similar agencies and organizations. Furthermore, managers 
attempt to leverage their IT investments to other operational and strategic activities across 
the entire enterprise. In this case, IT is the driver of strategic direction and organizational 
change instead of a reactive management tool. 

GAO uses this framework to assess an agency’s IT management processes and as 
a guideline for organizational improvement. The report outlines the purpose, 
prerequisites, and the activities involved in each of the critical processes and then 
describes how an agency may transition from one stage to the next. 

This useful framework will be further developed in this thesis to examine the 
critical processes necessary to address all three phases of the IT investment cycle and, 
from this, to build a model which can be used to assess an agency’s IT management 
processes. All of the critical processes in Stages Two, Three, Four, and Five are required 
to accomplish this. The critical processes of benchmarking and IT-driven strategic 
business change in Stage Five bring the agency to the point where IT investments drive 
the strategic direction of the organization. This chapter will expand this idea and 


27 




describe how to use the ITIM framework as an assessment tool for the IT management 
policy of a government agency. 

2. Using the ITIM Framework to Build Agency Policy 
GAO’s ITIM framework lays the groimdwork for evaluating IT management by 
addressing each phase of the investment cycle (Selection, Control, and Evaluate) in each 
stage of maturity for an agency. Figure 4 shows the three phases of the investment cycle 
and the proposed critical processes that were discussed above and included in Stages 
Two, Three, Four, and Five of Figure 3. 



Figure 4. Linking ITIM Critical Processes to the Investment Phases 

Once the critical processes are identified with the relevant phases of the 
investment cycle, they can be applied to the Dimensions of the IT Investment Evaluation 
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Approach outlined in Chapter II. This comprehensive three-dimensional representation 
of the agency’s IT management policy is depicted in Figure 5. 
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Figure 5. The Dimensions of an Agency’s IT Management Policy 


The agency can identify the processes in its IT management policy and compare 
them to the critical processes in the framework of Figure 5 and evaluate each for the 
necessary attributes of repeatability, efficiency, and completeness. 

The remainder of this chapter will describe the purpose of and activities required 

in each of these critical processes for each investment phase. 

B. THE SELECTION PHASE 

1. IT Investment Board Operation 

The IT investment board is the formal body designated by agency policy to make 
decisions concerning project selection, modifications to projects, and funding levels. IT 
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investment board membership and an agency-specific process guide (policies and 
procedures that define board operations) are an essential part of an agency’s IT 
management policy. 

Both senior line managers of major functions or operations and IT personnel must 
be represented in the board membership. The size of the board depends on the size and 
needs of the specific organization. The members of the board must be trained to 
understand their duties on the board and these duties must also be outlined in agency 
policy to ensure both repeatability and ease of review by agency administrators and 
outside agencies. 

The board s process guide must identify specific events where decisions must 
be made by the board. An equally important section of this guide outlines the 
environment in which the agency operates in terms of legal constraints or requirements 
and agreements with other agencies that must be honored, and how these affect the 
different decisions that must be made. This process guide will be the foundation of many 
of the other critical processes since it will define the format of deliverables provided to 
this board for use in decision-making. 

Agency commitment to the board must exist in two important areas. The 
agency’s top executives must provide strong leadership in defining the board’s 
responsibilities and membership and they must support and carry out board decisions. 
There may be exceptions where top administrators act contrary to investment board 
decisions, but these occurrences will be rare when the policies and procedures of the 
investment board are properly aligned with agency missions and objectives. 
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2. Business Needs Identification for IT Projects 

In order to meet the requirements of CCA and ensure all IT investments help the 
agency meet its mission goals, each investment must be tracked back to the agency 
mission(s) that it addresses. 

One way of connecting IT projects with the agency’s missions is by use of the 
“balanced scorecard.” The balanced scorecard can be used to express the vision and 
missions of the agency in terms of the value seen by its different stakeholders. Once the 
value seen by stakeholders is described, the IT investment(s) that best serve these 
stakeholders can be chosen and the performance measures to be used to control projects 
can be identified. 

Figure 6 outlines a tool that can be used to link the agency’s strategy to the 
various perspectives that will perceive success or failure. The critical success factors 
must be determined from each perspective as well as the IT investments that address 
these needs. Each perspective (financial, customers, internal business, and innovation 
and learning) will have different critical success factors and projects that address these 
factors. Performance measures are selected to track the investments’ progress towards 
satisfying the respective critical success factors. 

Once these connections between missions, stakeholders’ needs, investments, and 
performance measures are made, existing investments can be matched to their respective 
mission(s) and future investments can be fit into this existing scorecard to ensure a 
mission match exits. A record of investment mission matching must be maintained with 
investment records so this match can be evaluated in future reviews. 
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Figure 6. A Framework to Link IT Projects to Strategy After Ref. [7] 


3. Portfolio Selection Criteria Definition 

The portfolio selection criteria definition is the written policy and procedures for 
creating and modifying the minimum acceptable attributes an IT investment must display 
to be considered for addition to the agency’s IT portfolio. Developing an IT investment 
portfolio involves defining appropriate IT investment cost, benefit, schedule, and risk 
(CBSR) criteria to ensure the agency’s strategic goals, objectives, and missions will be 
satisfied by the selected investments (Ref [l]:p. 74). 

The portfolio selection criteria must be distributed throughout the organization so 
that the minimum expectations of potential investments are known and understood by 
managers and IT personnel. The portfolio selection criteria definition is the formal 
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declaration of the worst benefit/cost/schedule/risk relationship of an investment that will 
be considered by the agency for funding. 

Since the definition can change if the agency’s priorities or the environment the 
agency operates in changes, agency policy must also determine at what time interval the 
selection criteria must be reviewed to verify it is continuing to help the agency match its 
IT investments to its mission needs. 

4. Investment Analysis 

Investment analysis includes the data collected to support CBSR reports for 
investments and the use of the data by the IT investment board to assess and rank 
potential as well as current investments. Initially, for new investments, the analysis will 
provide an assessment of the agency’s best estimate of the future CBSR performance of 
the investment. As an investment is deployed, the analysis will track the realized costs, 
benefits, and risks so that proper attention is applied to the investment in the Control 
phase. 

Agency policy must delineate who performs the analysis and outline the 
procedure for conducting the data collection and presentation to the board. The policy 
must also describe how the IT investment board assesses the reports with respect to the 
portfolio selection criteria and prioritizes its full portfolio of IT investments. 

The investment analysis process provides the data that allows the investment 
board to select investments for consideration, rank the investments prior to allocation of 
funds, and track the performance measures of the investment during its control phase. 
The CBSR must be formalized so that it can be applied uniformly to all projects under 
consideration by the agency. 
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The information economics concept described in Chapter II is a usefiil way to 
combine the tangible benefits and costs with the intangibles. Thresholds are set on the 
scores from the information economics to define the portfolio selection criteria definition 
explained above. The CBSR will therefore be initially done for a potential investment to 
support its screening and selection, allow for tracking of the investment in the Control 
phase, and be scrutinized in the Evaluation phase to determine how the process can be 
improved. 

5. Proposal Selection 

The purpose of the proposal selection process is to establish a structured process 
to select new IT proposals. It is tied tightly to the portfolio selection criteria definition 
and investment analysis processes since these two provide the data input and minimiim 
requirements for new proposals to be considered for development. 

In order to accomplish the proposal selection, an individual manager should be 
assigned responsibility for this process. This manager’s responsibilities will include 
ensuring the staff imderstands the key events in the selection process, collecting all new 
proposals, and presenting all proposals umformly to the investment review board. 

In order to present all proposals umformly, there must be a formalized procedure 
to present the proposals in terms of their mission match and CBSR reports. From this 
presentation the investment board can select investments and allocate funds based on 
maximizing agency mission benefits while controlling risk and costs. The procedure 
must be flexible enough to allow improvement, but formal enough to provide clear 
instructions to the board. The output of this selection process is the recommendations 
from the IT investment board as to how the agency should allocate funds and develop the 
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agency’s IT portfolio. If the process is managed to properly address agency priorities, the 
agency management decisions should rarely be different from investment board 
recommendations. 

6. Portfolio Development 

The portfolio development process takes the recommended new proposals from 
the proposal selection process, including the agency’s current IT investments, and 
optimizes the IT portfolio with regard to mission accomplishment, costs, returns, and 
risk. 

In “Assessing Risks and Returns: A Guide for Evaluating Federal Agencies’ IT 
Investment Decision-making”, GAO explains a technique to display expenditures on IT 
investments in various stages of development [Ref 2]. All investments are assigned to a 
portfolio category. These categories include: research, development, operational, and 
enhancements. The “research” category contains investments under consideration for 
future use by the agency and being studied to gain a better understanding of costs and 
benefits associated with the investment. Investments in the “deployment” category have 
been selected for use by the agency, but are not yet fully operational. The “operational” 
category contains the IT investments currently fully deployed and in use by the agency. 
The “enhancements” category consists of upgrades to operational IT investments that 
have not yet been fully deployed. Figure 7 shows one way to display agency IT 
expenditures by category, including the percent breakdown of total expenditures within 
each year. The investments should also be displayed to connect them to the agency 
mission(s) they address 
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Figure 7. Example of IT Expenditure by Category After Ref. [2] 

By analyzing the trend of allocations in each category, the agency can ensure that 
current operations are not using resources needed for future research and develop projects 
so that the agency remains competitive and effective in IT. Likewise, by relating the 
projects in each category to the agency missions they support, the agency can begin to 
develop a succession plan for aging or high cost investments. 

Figure 8 displays a method of tracking the portion of each category (operations, 
enhancements, development, and research) that is being allocated to each mission 
addressed by the IT portfolio for a particular year. Managers will be alerted to missions 
that have increasing operational costs and no research or development on the potential 
replacements when this figure is expanded to show mission vs. category spending over 
several years. 
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Figure 8. Example of Mission Support by IT Budget Category for One Year 

7. Systems and Technology Succession Management 

This process develops a schedule for the replacement of investments or for 
enhancements to current systems as a form of technology refreshment. The data 
displayed in Figure 8, if expanded over several years, can be useful in displaying the 
trends of past spending and determining the past lifetimes of specific projects. The 
lifetime of a project can be driven by different factors such as the useful life of the 
product (i.e., laser printers), suspension of support by the vendor, or the rate of upgrades 
available in the market (i.e., office suite software or desktop computers). 

By understanding the natural lifetime of IT products, an agency can plan research 
and development so that replacements or enhancements are budgeted for and ready for 
deployment when they are required to replace an aging or outdated project. 
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8. IT-Driven Strategic Business Change 

At the highest level of maturity, the agency creates and maintains a knowledge 
base of state-of-the-art IT products and technologies. From this base, IT projects with the 
ability to strategically change the agency are identified. Strategic changes to the agency’s 
business practices are then planned for and implemented based on the identified IT 
capabilities in this base of knowledge. 

This requires the agency to constantly research cutting edge IT capabilities that 
may not initially appear to “fit” into the agency’s IT portfolio. When capabilities are 
identified that can help the agency strategically change the approach to a specific mission 
or consolidate the efforts previously requiring multiple investments, the new technology 
is developed and deployed. 

C. THE CONTROL PHASE 

1. IT Asset Tracking 

The most important process of the Control phase is tracking IT assets. The IT 
investments must be tracked by an inventory that is maintained in accordance with a 
formal agency policy. The inventory consists of a record of investments that have been 
selected as well as a record of the potential investments that were rejected or deferred. 
Included in the inventoiy records are a history of each investment’s current CBSR report 
and the expectations of the investment, including any history of changes to the 
expectations. 

The inventory record provides a common location for the documentation needed 
by managers and the IT investment board for various processes in the Selection and 
Control phases. The historical records of past investments provide a database that can be 
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accessed to assist in assessments and future investment selections. The inventory records 
of this process are required by the IT investment board to complete a thorough selection 
process ensuring all investments, potential and current, are considered. 

2. IT Project Oversight 

IT project oversight combined with portfolio performance oversight is at the 
center of the Control phase. The IT project oversight process maintains the most current 
cost and schedule data for each investment in the asset inventory records for review by 
project managers and the investment review board. 

The comparison of current cost and schedule with the expected results is the basic 
tool the managers and IT investment board should use to track investments under 
development. This will be further enhanced in the portfolio performance oversight 
process when the realized benefits and risk management are taken into account. 

During periodic review of projects by the IT investment board, investments not 
meeting cost and/or schedule expectations will imdergo further review. The deeper 
review will examine any corrective actions taken by project managers and determine if 
the investment should be modified or cancelled. The documentation of the project 
managers’ reviews and corrective actions, as Well as those by the IT investment board, 
are critical to the IT management process so that lessons can be learned from the 
problems and successes encountered. This history must be maintained in the IT asset 
tracking records. 

3. Portfolio Performance Oversight 

The process of portfolio performance oversight enhances IT project oversight by 
adding the benefits and risks of the projects to the cost and schedule comparisons. The 
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benefits and risks include many intangibles that can be addressed with the information 
economics concept addressed in Chapter II. In order for the uniform and repeatable 
application of information economics, agency policy must be very specific about the 
measurements or assessments made and the score associated with each. This will vary 
between agencies and will change over time in any agency as its missions or tasks change 
with time. 

Like the project oversight process, investments not meeting CBSR expectations 
are identified for deeper investigation. The investigations should determine if any 
corrective action is required, but should focus on determining the root cause for not 
meeting expectations and produce an actions plan for the recovery or termination of the 
investment. Other possibilities are cancellation or the adjustment of investment 
expectations should earlier expectations be found unattainable or too costly and 
investment cancellation would be more damaging to agency mission accomplishment. 
The reports and records fi-om these investigations must be maintained with the asset 
inventory for ease of review and evaluation. 

D. THE EVALUATE PHASE 

1. Post-implementation Reviews (PIR) 

The Post-implementation Reviews (PIR) is the foundation of the Evaluate phase. 
The PIR is the process by which an investment’s performance is reviewed after it is fully 
deployed or cancelled. The IT investment board determines which investments require a 
PIR and when the PIR should be conducted. 

Agency policy must delineate who may conduct PIRs and what tr aining is 
required of personnel that conduct PIRs. Personnel assigned to the project may be able 
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to more efficiently conduct the review, but the agency may chose to have personnel not 
directly involved in the project conduct the PIR to minimize any bias in the PIR process. 
However, the availability of sufficiently trained and knowledgeable personnel is often a 
problem. 

The information required for a PIR is both quantitative and qualitative. When the 
asset inventory process is properly in place, the majority of data required for this review 
will be present there. In addition, the agency will probably obtain data fi'om surveying 
stakeholders and personnel assigned to maintain the investment. This data will be useful 
in evaluating how the future control of investments can be improved. 

The final output fi'om the PIR is a report containing: conclusions about the 
investment’s actual results; user satisfaction; a review of the assumptions made during 
the data analysis and project oversight processes; lessons learned fiom the project and 
management of the project; a review of PIRs fiom similar investments in the past; and 
recommendations for the management of future investments or changes to the procedures 
in the agency’s IT management policy to improve the management process. 

2. Feedback from PIRs 

The IT investment board reviews all PIRs. The board 'will examine the lessons 
learned, the validity of the assumptions made during the investment’s development, and 
the recommendations made to determine how the agency’s selection and control of IT 
investments can be improved. 

The feedback fiom PIRs is the primary means by which the IT management 
process is subjected to continual improvement and adapts to changes in the agency’s 
goals or environment. By using feedback and the lessons learned fiom past projects. 
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future investments can avoid suffering the same mistakes and the agency can quickly 
determine when processes that provided past successes are failing to fit into the current 
environment. 

3. Investment Process Benchmar kin g 

The ultimate goal in maturity, Stage Five, is to understand how the agency’s 
processes differ fi*om the best practices and develop a method to attain the best practice. 
In order to accomplish this, the agency must develop a method to measure baseline data 
from their own IT investment management processes. Once this data is collected, 
external and comparable best practices are benchmarked using the same method to 
measure internal baseline data. By comparing the two data sets, the agency then makes 

improvements to the IT management processes to approach the best practices available 
for survey. 

The advantage this process provides is more rapid improvement in internal 
processes when an external organization with more effective or efficient techniques is 
found. The standard Select/Control/Evaluate framework improves agency processes by 
providing feedback on a trial and error basis. Investment process benchmarking can 
improve the internal processes faster by providing the vision of the ideal processes and 
therefore a target to achieve. 

This chapter developed a model of critical processes required in each phase of the 
IT investment cycle for management of an agency’s IT investment portfolio. The 
processes were grouped together by the phase of the investment cycle they occur within 
and a brief explanation of the goals of each of the processes was explained. 
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The next chapter will examine how this model can be applied to IT management 
within a specific federal agency that is small in size but critical to the information 
gathering for the national security of the U.S. 
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IV. IT INVESTMENT MANAGEMENT BY THE WHITE HOUSE 

SITUATION SUPPORT STAFF 


A. INTRODUCTION 

A model for evaluating an organization’s IT management processes was presented 
in Chapter III. This model can be used to examine the processes in place in order to 
compare them to the critical processes required for an organization with a mature 
management plan. This model will be used below to assess the processes in use by a 
specific organization, the White House Situation Support Staff (WHSSS). 

WHSSS is tasked with the selection and maintenance of commxmications systems 
and computer intelligence systems for the White House Situation Room (WHSR), the 
National Security Council (NSC) staff, and other White House offices. The mission of 
the WHSR is to provide current intelligence and crisis support to the President, his 
National Security Advisor, and the NSC staff. In order to accomplish this mission, the 
WHSR is required to gather all information concerning worldwide news, threats, or 
conflicts, and disseminate this information in a useable format such that the President and 
the NSC always have the data they need to make informed decisions concerning the 
position of the United States of America. To collect, manage, and distribute this 
information, the WHSR must be outfitted with state-of-the-art information search, 
collection and management systems. WHSSS administratively reports to the Defense 
Information Systems Agency (DISA) and receives their budget authority via the DISA 
budget. However WHSSS operates quite independently from DISA. 
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Operationally, WHSSS reports to the President’s National Security Advisor and 
through this connection, reaches its customer base of NSC directorates and White House 
offices. 

The planning, purchasing, operation, and maintenance of WHSSS’s IT equipment 
is managed by WHSSS’s Systems and Technical Planning Directorate. The Systems and 
Technical Planning Directorate is made up of about 36 persons, including military 
personnel, civilian employees, and civilian contractors. Procurement funds account for 
about one third of WHSSS’s annual budget of about $6 million dollars and when 
combined with Operations and Maintenance for major projects, flmding for IT projects 
account for over half of the obligated flmds. 

The remainder of this chapter will discuss the processes used by WHSSS to manage its 
IT investments and compare this policy with the model developed in Chapter III. The IT 
investment cycle for the WHSSS along with the critical processes (described in Chapter 
III) are displayed in Figure 9. Note that the critical processes for the WHSSS are 
somewhat different from those discussed in Chapter III. These will be explained in the 
subsequent sections for the Selection, Control, and Evaluate phases. 
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Figure 9. IT Management Policy at WHSSS 


B. SELECTION PHASE PROCESSES 

1. Introduction 

The strategic planning process at the WHSR has been relatively informal due to 
the dynamic mission of the agency. The critical nature of the tasks and operations of the 
agency helps unify the persoimel in a common objective and ease communication among 
the various decision-makers. The lack of a formal strategic plan has led WHSSS to adopt 
an IT selection process as described above and driven by “technology refreshments” or 
“innovations” instead of the portfolio approach described in Chapters II and III. 

WHSSS is in the unique position of having to replace significant amounts of 
equipment during the changeover of Presidential administrations due to requirements of 
the Presidential Records Act of 1978. This law changed the legal ownership of 
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Presidential records from private to public as well as set guidelines for the restricted 
public access to these records. The law defined Presidential records as: 

Documentary materials, or a reasonably segregable portion thereof, 
created or received by the President, his immediate staff, or a unit or 
individual or the Executive Office of the President whose function is to 
advise and assist the President, in the course of conducting activities 
which relate to or have an effect upon the carrying out of the 
constitutional, statutory, or other official or ceremonial duties of the 
President [44 U.S.C. 13 2201(2)]. 

This requires the storage of each paper and electronic record resulting from the 
President and Vice President’s official duties unless the Archivist of the United States 
approves of a specific item’s disposal. The scope of this law is quite far-reaching. 

In order to address the electronic records encompassed by the law, all forms of 
electronic storage need to be examined. This includes the hard drives and other storage 
media of all computers used by the White House and NSC, including the media contained 
in the servers of the computer networks used by all of the offices. While the media 
addressed by the law is of an administrative nature, the systems affected include 
administrative as well as mission critical IT investments. 

WHSSS uses this requirement along with the operational experience of equipment 
lifetimes to synchronize technology refreshment with administration changeovers of each 
new president as a form of cost avoidance. Since training must be provided for incoming 
personnel, WHSSS has avoided the cost of training twice by comb ining the technology 
refreshment with the time period of largest personnel turnover. A more intangible cost, 
but one still avoided, is the additional maintenance cost that would be associated with 
operating equipment for a longer period of time (approaching the end of its useful life) 

which may lead to failures due to age and wear. 
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2. Systems and Technology Succession Management 

The “technology refreshment schedule” is the driving factor for WHSSS’s 
Selection phase. That is, the selection process is “periodic” based on the technology 
refreshment schedule instead of the budget year as normally found in organizations. 
When an IT investment is selected for deployment, its total life cycle costs are budgeted 
in the WHSSS Program Objectives Memorandum and are reviewed annually during 
WHSSS’s budget estimate submission. 

When replacements of IT systems and hardware are scheduled for the next fiscal 
year, a team with members from each department within the directorate is formed to 
choose the new system. Potential replacements considered consist of upgrades to the 
current hardware (e.g., sophisticated workstations) or deployment of a completely new 
system (e.g., document management software). The team has the authority to evaluate 
the potential replacements and recommend the specific purchase to the director of the 
WHSR. The team membership will be slightly different for each purchase. This differs 
fi-om the investment board concept introduced in Chapter III. The investment board is 
charged with making decisions concerning the IT portfolio of an agency. WHSSS’s 
teams merely select the replacement from a list of alternatives for a system when the 
replacement is scheduled for technology refreshment. 

Team members include individuals from the following five departments: 
Financial Management and Acquisition, Operations, Research and Development, 
Maintenance, and User Support. A project manager is chosen from the Research and 
Development department to lead the team. 
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The input from the Research and Development department includes presenting 
state-of-the-art projects that have been researched for use by the agency and identified as 
having the potential to strategically change the manner the agency accomplishes the 
mission being addressed. These are the elements of the IT-Driven Strategic Business 
Change Process and this is how WHSSS moves technologies from research and 
development to deployment. Likewise, input from the User Support Department brings 
elements from the Business Needs Identification Process to the team by addressing 
customer needs and desires. A technology refreshment schedule for the next ten years is 
published annually and the out-years are evaluated on an annual basis or when Research 
and Development produces options that need to be addressed due to mission concerns. 

By using the technology refreshment schedule and making efforts to incorporate 
IT-driven strategic business change, WHSSS makes investment retirement decisions 
based on a combination of the technology replacement and implementation of 
investments from the Research and Development department. 

The selection of the replacement system could be enhanced by expanding the 
comparison beyond the measures of cost and schedule to include measures of mission 
benefit and risk analysis. This would require developing and utilizing an information 
economics scorecard as described in Chapter II. 

3. Business Needs Identification for IT Projects 

The goal of identifying the business needs associated with IT projects is to assist 
the agency in identifying the performance measure to use in ranking the potential 
investments and track the investment in the Control phase. WHSSS uses input from the 
User Support department to address the users needs and desires, but no formal connection 


50 




is made (as with the balanced scorecard technique described in Chapter III) to formally 
describe the critical success factors of WHSSS’s stakeholder’s (financial, customers, 
internal business, and iimovation and learning) perspectives and the performance 
measures that can measure progress in achieving success. 

4. Investment Analysis 

The analysis of new IT investments to replace current systems is based strictly on 
a cost-benefit analysis. There was no evidence from personal interviews with team 
members involved in a recent selection process that resource limits were considered. 
Instead, the most current technologies weighed against user needs and current hardware 
architecture (that would not be replaced) were compared based solely on costs and 
perceived needs of the users and maintainers. This in no way implies that funding is 
unlimited, but that the agency personnel seem to be very effective in restricting 
investments to those projects that are mission essential. 

The selection process starts from a needs definition. The needs definition is 
derived from any hardware architecture requirements to support interoperability with 
other WHSSS hardware or hardware of external agencies and the perceived needs of the 
customer. Several alternatives are identified and evaluated based on the costs and 
schedule requirements. By following this procedure for each system upgrade or 
replacement, WHSSS avoids the task of actually calculating an enhanced ROI and 
ranking the alternatives, but still selects the investment that would have the highest 
enhanced ROI, thereby meeting the intent behind the GAO guidance. 
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C. CONTROL PHASE PROCESSES 

1. IT Project Oversight 

Once a project is selected and enters development, the project manager helps 
develop the project’s budget and tracks the costs and schedule throughout the deployment 
of the investment. WHSSS has no current provision to address the benefits realized or to 
determine how the risks involved with the investment are changing or managed as 
addressed in the Portfolio Performance Oversight Process. The positive aspects of 
tracking risks and benefit realization are especially useful to managers to assess progress 
early in development and deployment when there is no equipment yet operational. 

An example that displays a weakness of addressing customer needs by support 
staff input instead of customer involvement, as well as the problem of not tracking 
benefits expected and realized, was displayed during a recent deployment of a software 
investment two years ago. The NSC staff is supplied a search engine to aid in sifting 
through the vast number of news wires to assist in their research for news items. The 
incumbent software needed to be replaced due to year 2000 compliance problems. The 
replacement software was chosen to correct the Y2K compliance requirement and 
increased versatility was expected, but the user interface differed greatly from the old 
software. Following implementation, the new software resulted in a flood of calls to the 
User Support Department and great user resistance and resentment towards the new 
investment. Involvement of the users in this software replacement would have prevented 
this bad experience. 

As a result of the above event, software upgrades or replacements are now 
implemented only after a “help manual” has been developed and users are trained on the 
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differences between the old and new systems. Additionally, the User Support 
Department deploys the investments in each NSC directorate separately and identifies a 
properly trained staff member in that directorate office as the point of contact to answer 
user questions until no longer needed. This method was used in the recent deployment of 
the Microsoft Office 2000 suite and users encountered very few problems. 

2. IT Asset Tracking 

All IT hardware inventory is tracked in a database by WHSSS. However, the 
goals of the IT Asset Tracking Process cannot be realized, because the database merely 
tracks the receipt and physical location of the equipment. There is no method currently 
available in the database to connect an entry for a piece of equipment to a specific 
investment. Operators in the Maintenance and Operations Departments have developed 
methods to track the equipment that they regularly deal with, but this is done in an ad hoc 
manner. 

There is a project under review that will replace the current database with a 
system that will identify individual pieces of hardware with their respective investment 
project as well as tie in lessons learned during the development (fi-om Research and 
Development Staff) and operation (fi-om the User Support Staff). This would greatly 
improve the IT Asset Tracking as well as improve the feedback fiom the lessons learned 
in the Evaluate phase. 

3. Modification or Cancellation of IT Projects 

“Continued investment analysis” (cost and schedule only) is conducted by the 
program manager during IT investment deployment to track the investment’s 
performance. No record of corrective actions taken is maintained during the deployment. 
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There are no formal procedures to report problems during the project’s deployment other 
than the thresholds at which the level of management required to add additional funds to 
the investment’s budget is specified. 

The use of the balanced scorecard, as described in Chapter III, would provide 
additional performance measures that could be tracked and used to determine when 
corrective action and additional managerial attention is required. 

D. EVALUATE PHASE PROCESSES 

There is currently no formal process to perform Post-Implementation Reviews or 
provide feedback from lessons learned so that future improvements can be made to the 
Selection and Control phases. Instead, individual WHSSS departments, especially the 
Research and Development staff, have attempted to gain the benefits of a Post- 
Implementation Review by developing a procedure to incorporate lessons learned into the 
next Selection and Control review of a given investment. This includes the “corporate 
knowledge” of individuals involved with the last technology refreshment. However, this 
knowledge is often difficult to locate since these vital people may not be involved in the 
next process or they may no longer be employed by WHSSS. 

An example of how the need for a PIR process recently became evident was with 
the last replacement of laser printers. Few personnel from the previous purchase decision 
were involved in the replacement process. The decision for the replacement printers was 
based mainly on the relationship with the current vendor and a replacement was selected 
from the most technologically advanced options based on cost and schedule of delivery. 
As a result of the options, the new printers had a larger footprint and did not fit in some 
office locations. Additionally, a technical difference existed in the manner the printer 
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processed templates in the current software suite. The User Support Department 
struggled with answering user questions and developing new templates. The User 
Support and program manager assigned with selecting laser printers now have a long list 
of lessons learned to help in the next selection process, but these exist in individual files, 
not in organizational records. The planned replacement for the IT Asset Tracking System 
should help alleviate this problem. 

The purpose of the Evaluate phase is to examine the lessons learned from each 
specific project and use these to adjust the processes in the Selection and Control phases 
to continually improve the agency’s management of its IT investments. While WHSSS 
makes an effort to apply lessons learned to the next decision process, there is no process 
in place to modify the processes to incorporate these lessons learned. To address the 
lessons learned issue, WHSSS requires two changes to current practices. First, the 
processes used need to be formalized in some manner so that they are readily available 
for use and modification. Second, a PIR process needs to be formalized to allow the 
lessons learned to improve the agency’s processes and make the lessons learned more 
accessible. The use of the PIR would also provide a better method of assessing customer 
satisfaction with the investments instead of relying on User Support department records 
of trouble calls. 

This chapter has used the IT investment management concepts from earlier 
chapters to evaluate the one specific federal agency’s practices. It appears that this 
agency, through a combination of informal and formal practices does comply with the 
spirit of the Clinger-Cohen Act’s intent. That is, to assess the effectiveness of IT 
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management processes. In addition, WHSSS personnel are already making changes and 
taking action to improve those IT management practices that fall short of the objective. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

1. Evaluating IT Management 

By the passage of the Clinger-Cohen Act and the active involvement of the 
General Accounting Office, an oversight arm of the Congress, in IT issues, the U.S. 
Congress has signaled the federal agencies that management practices for IT investments 
must improve. This is understandable given the ever-increasing expenditures on IT 
projects \vithin the federal government. This thesis has attempted to address the major 
concerns of both the Congress and the managers responsible for implementing new IT 
management practices. 

To build a basic outline for an agency’s IT investment management policy, this 
research first utilized the GAO’s three phases of the IT investment cycle as the major 
themes for the model. These phases are Select, Control, and Evaluate. In addition, the 
critical processes outlined by GAO in the maturity stage model (displayed in Figure 3) 
provided new concepts for addressing questions of efficiency and effectiveness within 
each of the IT investment cycle phases. The maturity stages was an important addition 
because it sharpens the focus on an organization that is positioned at a specific level of IT 
management, ranging from simple project oriented organizations to those in which IT is 
an integral part of its strategy and mission.. 

The complete model was discussed in Chapter III, including the critical processes 
required by each phase of the investment cycle. Then in Chapter IV the recommended 
IT investment management model was used as the basis for comparing the IT 
management processes that were in place in a specific agency (displayed in Figure 9). 
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This management tool helped identify the differences between the agency’s processes in 
use and those of the ideal agency with fully mature IT investment management processes. 
The major contribution of the model is that it allows an agency to quickly assess what 
improvements need to be made to its processes and what new processes should be 
investigated for proper fit in the agency’s business environment and possible 
implementation. 

At the highest maturity level of IT technology awareness, the IT-Driven Strategic 
Business Change process is the vision of using IT to change the focus or direction of an 
agency to gain strategic advantage. This is one of the goals of CCA and these gains can 
be realized on the battlefield or in the business operations of the government. 

When developing IT investment management processes for an agency or 
assessing the processes in use one essential point must be addressed. While there is a 
common foimdation on which to build an agency’s IT management policies, each agency 
needs to evaluate the specific operations they face and choose the implementations that 
best fits their agency and the environment they operate in. This is both recognized by 
GAO [Ref. 1 :pp.4, 5] and displayed in the example of the White House Situation Support 
Staff in Chapter IV. 

2. WHSSS Approach to IT Management Processes 

The organization selected as a site for applying the recommended IT management 
model was the White House Situation Support Staff While very small in comparison to 
other federal agencies, this organization can be characterized as IT intensive. That is, the 
critical mission of the agency in U.S. national security matters requires use of the best 
information technology and communication systems. 
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As pointed out in Chapter IV, WHSSS has chosen a management technique that 
balances the requirements of periodic technology refreshments with that of purchases 
associated with large personnel turnovers and legal mandates. Presidential elections 
force personnel turnovers, which obviously are beyond the control of the WHSSS. 
Likewise, legal archive requirements place a heavy burden on IT for document 
management and storage. The technology succession plan they operate under assumes 
cost avoidance. Also, they manage the upgrade or replacement of systems to attempt to 
choose the system with the best return on investment without actually calculating a ROI. 

Without documenting the procedures they use, or starting from the requirements 
of CCA, the WHSSS have successfully used their own procedures to meet the intent of 
the CCA by striving to choose the product with a cost and lifetime that best fits the 
succession schedule that has been chosen for the environment they work within. Without 
documentation of the procedures used, the data used in decision-making, or a history of 
past decisions made, it is difficult to predict the repeatability, completeness, and 
efficiency of future IT investment cycles or explain past cycles. 

The cost avoidance theory of their current IT management technique along with 
the technology refreshment process that evaluates new abilities of maturing technologies 
and addresses the customer needs and desires may indeed be more efficient than 
developing the portfolio approach outlined in Chapter II. However, without a list of 
strategic goals and the connection of these to the elements of the Balanced Scorecard in 
Chapter III it is very difficult for WHSSS to develop performance measures to track the 
mission accomplishment provided by IT investments. Also, it is difficult to defend the 
current technique by comparing it to GAO and OMB’s best practices. 
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Once a strategic plan is developed and implemented, the current processes can be 
evaluated further and either be improved or replaced with a more efficient management 
technique. 

B. RECOMMENDATIONS 

1. Evaluating IT Management 

As noted in Chapter III, the maturity levels identified by GAO may be difficult to 
apply to an agency that has IT projects using processes in some or all of the maturity 
levels, but missing critical processes at the lower maturity stages. This would be 
expected for most agencies that have been managing IT projects in the past, and would 
result in a low maturity assessment despite the higher level processes in use. Using the 
model developed in Chapter III to address all the critical processes and measure the 
effectiveness of each process can be a much more useful technique to assess the IT 
management of an agency and allow them to address the weaknesses present in all phases 
of the investment cycle at one time. 

As mentioned in Chapter III, the efforts in the Department of the Navy to develop 
a information scorecard for screening and ranking potential investments is helpful for 
agencies in the Navy, but the guidance should be expanded to explain how the scorecard 
can be altered to meet a specific agency’s mission needs and business environment. This 
identifies two very important points; guidance must be given concerning how to screen 
and rank investments, but this guidance must be generic enough to apply to all agencies 
in the department and the guidance must include a description of how an agency can alter 
the scorecard to best fit the specific agency. 
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2 . 


WHSSS IT Management 


The vision of the Clinger-Cohen Act is to take the benefits of the Government 
Results and Performance Act and extend them to the IT investments of the government. 
Without a strategic plan and the focus of the organization’s direction on the goals set in 
the plan, it is difficult to measure the efficiency by which WHSSS manages its IT 
investments. Once a strategic plan is developed, WHSSS can use the balanced scorecard 
outlined in Chapter III to connect its mission to the critical success factors. When this 
connection is made, performance measures can be chosen to track the mission 
accomplishment of its IT investments purchased to address these missions. Additionally, 
the use of performance measures will provide WHSSS with a method to measure and 
display the improvement in mission accomplishment with each generation of technology 
refreshment. 

While many effective, and in some cases efficient, processes exist in WHSSS IT 
management, the most obvious flaw is that neither documentation of the techniques used 
in each process nor the data from previous decisions (including the decision made and the 
data used to make that decision) is readily available. Without a documentation of how 
these processes are conducted and history of the decisions made, the repeatability and 
completeness of the processes are difficult to predict since they depend on the presence of 
personnel knowledgeable of previous decisions. Likewise, the efficiency of future 
decisions cannot be predicted and the efficiency of past decisions are difficult to evaluate 
without a record of the data and techniques used to make the decision 

The most obvious improvement that can be made to WHSSS IT investment 
managements is displayed in the empty block in the Evaluation phase portion of Figure 9. 
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Without a fonnal Post Iinpleinentation Review process and a method for the review to 
provide feedback to improve future Selection and Control processes, WHSSS fails to 
address the second requirement of CCA. The processes in use must be continually 
improved to ensure inefficiencies of the past are discovered and prevented from occurring 
in the future. 

C. AREAS FOR FURTHER RESEARCH 

Two areas of research could improve the effectiveness of evaluating IT 
management processes. First, the maturity level model outlined by GAO may be 
improved by developing a model whose maturity levels address all phases of the 
investment cycle (Select, Control, Evaluate) and each maturity stage. For instance. Stage 
Two would consist of the processes necessary for simple application of Select, Control 
and Evaluate processes. All three phases would then gain more mature processes as the 
agency matured through each stage. This would be more useful than the current model 
since agencies generally have processes in place that exist in multiple maturity stages of 
the GAO model. This newer model would allow an agency to improve its Selection 
phase processes in Stage Two, while also improving its Control phase processes that may 
be at Stage Four. 

Second, there should be more research into the methods used by both corporations 
and government agencies to link IT investments to missions and techniques such as the 
balanced scorecard technique described in Chapter III. This would assist in the selection 
of performance measures for tracking IT investment performance, and help improve the 
Selection phase ranking and the methods used to track investments in the Control phase. 
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Additionally, WHSSS is a very unique agency in terms of the enviroimient it 
operates in. This somewhat limited the application of the model developed in Chapter 
III. Additional research that applies this model to other agencies, especially larger 
organizations with numerous divisions with different IT needs, is required to better test 
its ability to assess an agency’s IT management regardless of size and IT project 
complexity. 

Concerning IT investment management at WHSSS, once the strategic plan has 
been developed and implemented, a study of the improvements, if any, that were made to 
the agency’s IT management plan should be conducted. This can be based on the 
research of this thesis or based on the focus provided by the strategic plan. Such a study 
will aid WHSSS in both setting the baseline of their investment processes and 
implementing investment process benc hmark ing. 
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